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A. PREFACE 

As the educational rrform efforts of the eighties carried on into the nineties, the Office of 
Educational Research and Improvement launched a program of research studies which addressed 
various facets of educational reform. The research reported here is the result of one of 12 such 
studies initiated by that Office in 1991 . 

The focus of this four-year research project has been curriculum reform, with specific attention 
to the three areas of science education, mathematics education, and higher order thinking across 
the disciplines. 

Significant curriculum change is more than a curricular matter; it extends into most other facets 
of schooling, including teaching, learning and the culture of the school. Major change demands 
the attention of parents as well as the full range of school personnel. 

As past research--and this study-shows, educational reform is an ongoing process and seemingly 
never complete. It requires a major commitment over a long period of time. This study tells 
this story in considerable detail, including the nature of various reforms, the barriers to reform 
encountered, and the means by which challenges were met and positive change achieved. 

The report of this research is contained in three volumes: I. Findings and Conclusions, n. Case 
Studies, and m. Technical Appendix: Research Design and Methodology. This volume (III) 
contains a description of the research design and various facets of the research methodology, 
including sample selection, data collection and data analysis. For a full report of the cross-site 
analysis of the case studies see volume I. The reader wishing to read the case studies in their 
entirety is referred to volume n. 

This report is presented with the expectation that it will be helpful to others pursuing educational 
reform, whether they be policy-makers, practitioners, parents or researchers. With it go best 
wishes to all in the quest for improved education. 



Ronald D. Anderson 
Boulder, Colorado 
June 1995 
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E. RESEARCH DESIGN 



[NOTE: Some of the information in this section and the following one also is included in 
Volume I.] 

The central goal of this research was to conduct case studies of schools~or a department within 
a school-which were engaged in curriculum reform. These sites were to be ordinary school 
settings in the sense that they were not dependent upon large infusions of new funds or the 
intellectual resources of experts devoting l^e amounts of time to the site. They obviously 
could be under the influence of outside support in the form of funds and expertise, but the 
amount of such support would be within bou^ most schools could plausibly encounter through 
rather routine federal or state programs of support for educational change. 

A research plan was developed as the foimdation for this set of nine case studies-three each in 
mathematics, science and thinking across the disciplines~to be conducted during the 1992-93 and 
1993-94 school years. 

The Research Questions 

Among the prominent research questions are ones pertaining both to the substance of the reforms 
and the means by which the reforms were put in place. The substance of the reforms includes 
attention to both the content of the curriculum and the instruction by which students acquire it. 
The means by which the reforms are put in place includes particular attention to systems thinking 
and the overall patterns of reform activities. 

The substance of reform. The following questions are amongjhose addressed with respect to 
the curriculum reforms themselves. 

1. How are the sites defining their purposes and goals of reform with respect to 
students, teachers and the rest of the system? 

2. What changes have occurred in the content of instruction? 

3. What changes have occurred in the means of instruction, i.e., how are teachers 
fostering students’ ability to construct desired learning outcomes? 

4. What has been the impact on student learning and what can be inferred from positive 
results about various ways of teaching science, mathematics, and higher order thinking ? 

5. How "deep’ are the changes: i-e, have the beliefs of students, parents and teachers 
changed? 

6. To what extent and how is the learning and teaching of thinking skills being 
transferred across disciplines? 



The means of reform. Attention also is directed to how people got to where they are. This 
investigation includes looking for patterns of support throughout the system and examining how 
people monitor their progress toward desired goals. 

1 . What are the mechanisms for change? 

2. How do implementation efforts play out in classrooms; i.e. what happens in 
classrooms when teachers embrace the spirit of the reforms espoused by the various 
national groups? 

3. How are sites developing high quality content that meets the needs of their full range 
of smdents? 

4. V /are sites developing teachers’ abilities to use a constructivist way of teaching? 

5. What are the dynamics of change as viewed from a teaching and learning perspective 
among (a) students, (b) teachers and (c) the rest of the system? 

6. How are sites working out the trade-offs related to financial and expertise resources? 

7. To what extent has the process of reform been top-down, bottom-up, or some 
combination of the two? 

8. What are the means of system support? To what extent and in what ways are the 
system support strategies congruent with a constructivist view of teaching and learning? 

9. To what extent does the approach to change at the sites reflect systems thinking? 
Conceptual Framework 

Fully understanding a complex and dynamic situation requires examining it from a variety of 
perspectives. Looking at it fircm a variety of vantage points provides a fullness of understanding 
not possible when seen during a limited time q>an, through the eyes of only .selected categories 
of participants, or from the standpoint of selected interactions within the total dynamic setting. 

Many perspectives are built into the conceptual framewortt to ensure that field observations and 
interviews acquire data needed for a wide range of analyses. Coherent portrayals of the situation 
in each school later will require a limited number of analyses, but at the beginning a fairly broad 
set of perspectives is demanded, as described below. 

Time perspective. The conceptual framewoik for this research is based on looking at a "slice" 
of the present (portions of one academic year) to understand (1) the past from which it came, 
(2) the present (in terms of influences, results and dilemmas), and (3) the perceptions of future 



destinations held by the various people involved. As a result the research questions stated above 
must be expanded upon to give a form such as the following. 

• What are the past patterns of school practice from which the current 
practices emerged? 

• What is the nature of current school practice? 

• What is the Juture whicu the various actors envision as their intended 
destination? 

Influences, results and dilemmas. To understand more of how changes over time have and 
are occurring, it is necessary to e xamine the dynamics of the situation including the following. 

• What influences (e.g., pressures, supports or barriers) liave affected these 
reforms? 

• What have been the results of these reforms? 

• What dilemmas have arisen for the various actors in these reform efforts? 

Multiple dimensions. These questions about influences, results and dilemmas encompass at 
least the following four dimensions: 

• Personal: influences based in the knowledge and beliefs of the individual 
actors involved in the reform effort as these beliefs and knowledge relate 
to such matters as the discipline, teaching, learning, school and smdents. 

• Interactional: influences based in the interactions of people in classrooms 
or other settings. 

• Contextual: influences arising in the social, cultural, structural, 
organizational, political, or historical context. 

• Historical: influences arising from past events and their resulumt 
structures, patterns of interaction, and convictions of people. 

Interest group perspectives. What are the perspectives of the following at each site and how 
do they compare: students, teachers, administrators, policymakers, and parents? How does the 
reform as defined at each site map onto reform ideas as ^fined by (1) professional grrups at 
all levels, and (2) the public in general? 

Systems thinking. How can a systems approach to analyzing these reforms help in 
understanding the changes? 
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Subject-matter perspectire. Because of the importance of subject-matter considerations wiUiin 
this research, the research questions must be explored from both discipline and non-discipline 
specific perspectives. 

Critical components, the following guides to observation, questioning and analysis will be 
used:' 



• Student goals and expectations as exhibited in intended anr’ actual 
learning outcomes, 

• Teacher professionalism as reflected in professional development activities 
and participation in professional work such as curriculum development. 

• Vision as indicated by changes in school curricula or pedagogical 
practices, school routines or daily operations, and statements of vision. 

• Curriculum, instruction and assessment design including the degree of 
complementarity of these three. 

• Changing roles of students, teachers, administrators and parents, with 
attention to their responsibilities and the interrelationships of these roles. 

• Resource allocation, including both financial allocations and the 
expenditure of professional time. 

• Means of expanding the reform to additional individuals and groups, or 
portions of the curriculum. 

Research Context 

This research was pursued in the context of nine schools selected because they were successfully 
moving toward reformed education and were ordinary schools, not special settings with unusual 
resources. 
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F. METHODOLOGY 



This research employed a case smdy methodology using various qualitative techniques. The 
follovnng sections include a descripti m of the site selection, data collection, and data analysis. 

Site Selection 

The site selection process identified programs in nine schools, three each in mathematics, science 
and thinking skills across disciplines, and negotiated access to them for conducting case smdies 
of curriculum reform. Because of a temporary interruption in the contract funding due to 
congressional appropriations process, the site selection process actually took place in two parts 
for two different academic years. The nature of each search and its associated criteria are 
presented below. 

Initial site selection. The initial selection process involved a lengthy and intensive search which 
identified multiple sites for all three subject areas, based on several specific criteria. 

Selection critetia. Based on a review of the research literature on curricular reform in science, 
mathematics and thinking across disciplines, individual school sites were sought having the 
following, non-prioritized five characteristics: 

1 . Enabling all students to learn to think is an educational goal. 

r An important theme is a so-called constructivist approach to learning. 

3. A related theme regarding the curriculum goes undeiL the label: "less is more". 

4. Instead of the presentation of isolated facts, major attempts are made to focus on 
major themes of the subject matter and foster an integration of knowledge across 
disciplines. 

5. Constructivist learning requires constructivist approaches to teaching; rather than 
viewing students as passive recipients of information, teachers must focus on helping 
students construct understanding of concepts for themselves. 

Schools having several of these reform characteristics in place, and having evidence of positive 
outcomes for students were considered as case study sites. 

The search process. The search process consisted of two phases-a nomination phase and a 
contact/infonnation-gathering phase. 

Nominations. The search process began with the solicitation of nominations from researchers 
and kno . '-^geable practitioners across the U.S. For example, approximately 400 of the leading 
science education researchers were contacted at a general session of the annual convention of 
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the Naiional Association for Research in Science Education and given written no mina tion forms. 
A similar approach was used at a meeting of researchers at a conference of the National Council 
of Teachers of Mathematics. Letters were sent to leading scholars familiar with higher thinking 
skills programs in schools. 

In addition specialized contacts in some areas were employed. In science, the leaders of the two 
major reform movements—the National Science Teachers Association (f^STA) Scope, Sequence 
and Coordination (SS&C) Project and the American Association for the Advancement of 
Science’s (AAAS) Project 2061~were asked for suggestions of schools where these reforms were 
in place and underway. Several SS&C sites were proposed but Project 2061 leaders did not 
believe any of their sites were not sufficiently operative to qualify for the study at that time. 
Major curriculum development organizations were asked to suggest sites as were members of 
the Curriculum Reform Project Advisory Group. The selection of mathematics sites also 
involved nominations from Woodrow Wilson Foundation mathematics participants and 
instructors. 

Contact/information. Most nominations were followed up with telephone conversations with 
the nominators. Based on these discussions, decisions were made as to which nominations were 
worthy of follow-up by direct contact with the schools. Key people at the nominated schools 
were then contacted by phone to get additional information. Written exchange of information 
followed. Information sought from sites included indicators of student learning outcomes. After 
sites were identified and preliminary contact made, a list ^f the sites was forwarded to OERI for 
approval. 

Geographical location. The search process was directed at sites across tlie U.S. with no 
selection criteria regarding geographic location. The nine sites are located in seven different 
states, widely distributed across the country. 

First round sites 

Thinking across disciplines. School in a suburban district in the western U.S. 

• 9-12 high school 

• 1350 students with about 20% minority population 

• middle to lower class socio-ecorximic status 

• been in operation 4 years (at time the site was proposed for inclusion) 

• focus on students in core curriculum taken by dl 10th grade students 

• core curriculum integrates science, social studies, English and some other 

subjects 

• 21 teachers involved in the core 

• entire school attempting to emphasize active involvement of students in learning, 
team teaching and higher order thinking 
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Mathematics. School in a suburban district in the western U.S. 



• new suburban high school 

• initiative for reform has come from the mathematics faculty of the school 

• beginning third year of operating a 3-year integrated mathematics program - 
Interactive Mathematics Program (IMP) developed at Lawrence Hall of Science 
and San Francisco University 

• also operate a traditional mathematics program and have comparative data 
Science. School in a suburban district outside a large California city 

• a Scope, Sequence and Coordination Project (NSTA) school 

• program is tied to the California "State Framework" 

• faculty includes leaders in developing science reform in the state 

• teaching coordinated science-a three year integrated science cuniculum 

Second round site selection. In round two, six additional case studies were initiated. 
Information and understanding developed in conducting the fust three studies informed a new 
round of site selection criteria. Several schools considered in the fu^t round were not considered 
during the second round of site selection for several reasons: lack of interest on the part of the 
school to participate at this later date, turn-over of administration with whom access had initially 
been negotiated, and similarities between sites already studied and those remaining to be studied. 
When additional criteria were added to broaden the generalizability of the group of case studies 
and the cross-site analysis, others of the remaining six original sites were eliminated from 
consideration. 

Ilie additional search process. The additional search process again consisted of two phases- 
nominations and contact/information- gathering. 

Nominations. The solicitation of additional site nominations for case studies was conducted 
through a letter campaign to selected resource people whose knowledge of the particular 
curricular area and schools was considered to he outstanding. Follow-up phones calls were 
made to all the sources. 

Contact/information. Individual contact people at the nominated schools were contacted by 
telephone to ascertain the school’s program in the area of interest-thinking skills, mathematics, 
or science-student demographics, and school reform characteristics. The contact person then 
sent descriptive information about the program and available data about student learning to the 
Curriculum Reform Project. 
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Second round sites. 



Thinking across disciplines. School in a suburban district on the West Coast 

• 6-8 nuddle school 

• 1200 shulent with about 56% minority students (46% Hispanic, 8% African- 
American, 2% other) 

• two years using High Success Network’s Transformational Outcome-Based 
Education measured by student achievement of locally defmed exit outcomes 

• thinking skills program centered around Universal Dimensions or Spheres of 
Living 

• exit outcomes were related to world of woric, community and family 

School in an urban district in the Southeast 

• 9-12 high school 

• 12(X) students with 27% African-American students 

• 7 years of restructuring, 5 years affiliation with Coalition of Essential Schools 

• focus of study is locally developed 1 1th grade two-hour interdisciplinary course 
in history/English 

Mathematics. School in an urban district in the northwest 

• 9-12 high school 

• 1600 students with 25% Hispanic students 

s long history of math reform paralleling national trends 

• reform driven by individual teachers using existing materials available 
commercially-i.e., The North Carolina Materials, The University of Chicago 
School Math Program, and one program. The Integrated Science-Math Course, 
developed in-house 

School in an urban district in the Midwest 

•’ 9-12 high school 

• 1700 students with largest miimrity population Native American 

•< middle to lower-middle class 

• reform predicated on a State Systemic Initiative grant 

• teachers active in various aspects of SSI reform over three years-writing, 
piloting, using, assessing curriculum and teacher training 

Science. School in a district straddling urban, suburban and rural boundaries along East Coast 

•' 6-8 middle school 

• 50% minority student population 
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• reform involves implementaton of a state developed Scope Sequence and 
Coordination program 

• site-based management contributed to school’s participation 

• reform in existence for 2 years 

School in a district serving a military base, rural agricultural area and small city in the Midwest 

• 6-8 middle school 

• 700 students with 38% African-American, 9 % Hispanic, 5 % Asian-American or 
Pacific Islander and 1 % American Indian or Alasl^ Native 

• reform district initiated implementation of commercial program developed by an 
NSF-funded curriculum project 

• reform in effect 2 years 

A comparison of these schools for length of time in reform and source/impetus for reform is 
given in the table 1 below. A comparison of all nine sites for grade level impacted, geographic 
area, type of district-rural, suburban or urban, minority population and socio-economic status 
of the community-appears in table 2 below. 



Table 1. Comparison of All Sites for Length of Reform Operation and Source of Reform 



Subject Area 



Time of Reform 
Operation 



Developer of Reform 



Thinking skills 
Thinking skills 2 
Thinking skills 3 
Mathematics 1 
Mathematics 2 



S years 
2 years 
7 years 
4 years 
7 years 



Teachers in the school 
Commercial enterprise 
Local teachers 
National developer 
Commerical textbooks and 
Teacher development 
State Systemic Initiative 
Local teachers in a state 
and national context 
State developed SS&C 
District initiated use of 
commerical program 



Mathematics 3 
Science 1 



3 years 
3 years 



Science 2 
Science 3 



2 years 
2 years 



Table 2. Comparison of Nine Sites for Seyeral Geographic and Social Considerations, 



Subject 


Grades 


Geographic 

Area 


Thinkings 
kills 1 


10th 


West 


Thinking 
skills 2 


6-8 


Southern 
west coast 


Thinking 
skills 3 


11th 


Southeast 


Math 1 


9-11 


West 


Math 2 


9-12 


Pacific 

northwest 


Math 3 


9-11 


Midwest 


Science 1 


9-11 


Southern 

west 

coast 


Science 2 


6-8 


Southern 

east 

coast 


Science 3 


6-8 


Midwest 



Type of 
Di^ct 


Minority 

Population 


SES 


Suburban 


20% 


Middle to 
low 


Suburban 


56% 


Middle 


Urban 


27% 


Middle 
to low 


Suburban 


low 


High to 
middle 


Urban 


25% 


Middle 
to low 


Urban 


low 


Middle 
to low 


Suburban 


- 


Middle 
to low 


Combina- 
tion of 
all 3 


50% 


Middle 
to low 


Rural to 
urban 


52% 


Middle 
to low 



Data Collection 



Although each case study was conducted by a different researcher, the data collection process 
across the nine case study sites was similar. This commonality of approach was reinforced by 
periodic meetings of the researchers during the research period to discuss data collection and 
anal ysis issues. In general, data were gathered primarily in the form of 1) classroom 
observations, 2) interviews with students, teachers, administrators and other key personnel, and 
3) documents fiom the reform sites. 

Classroom observations. Classroom observations at all sites generally focused on behaviors, 
interactions, and conversations of both students and teachers. These data reflected key 
information about 1) the reform effort as it played out in the classroom setting, 2) student 
l earning , 3) teacher l earning and teaching, and 4) assessment. Different sites offered 
opportunities for more or less scratiny of each of these aspects of the reform. In some sites 
classroom interactions, such as presentations and group work were audio- and/or video-taped for 
future analysis. In some sites hallways were settings for informal discussions with teachers 
and/or students. One set of "classroom" observations even included a trip to the off-campus site 
where the class convened. For the most part, classroom observations were the most uniform 
in practice across the nine sites. For efficiency and clarity, however, se^'eral case smdies 
foctised on a few teachers rather than all the teachers implementing the reform. 

Interviews. Universally both students and teachers were interviewed to elicit their perspectives 
about the reform and its impact on them as participants in it. In most cases, building level 
administrators were also interviewed to provide a school level view of the reform. Where 
appropriate, other school administrators, district administrators, program developers and 
university support personnel and trainers were interviewed to provide a wider perspective of the 
change process on the teachers, students, and other impacted groups and to gain an 
understanding of the overall context for the reform. 

The number of smdents and teachers interviewed was site specific and dependent upon the 
ffamewoik of the case and its level of focus within the reform. Generally an effiirt was made 
for equal representation across gender, academic achievement, racc/ethnicity and tenure within 
the reform. At one site a cadre of IS students was chosen by teachers to fit the research criteria 
matrix of academic needs and potential with modes of eiqpression-specifically written, oral or 
artistic-for both sanqiles of student work and interviews. In a number of instances, students 
ffi>m the previous year’s program were interviewed to provide a sager viewpoint of the reform. 
Those teachers whose classrooms were observed most often, also were interviewed more 
extensively than other teachers. 

Dociunent collection. While the distinctive character of each reform site generated a unique 
set of documents necessary to describe the nature and process of each reform, many similar 
types of documents were collected at most sites. These documents included curricular materials, 
student work, and contextual information. Curricular materials demonstrated the individual 
character of the reform and teacher role. Student work represented examples of both student 
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learning and assessment formats. Contextual information was usually collected in the form of 
documents from building, district or state levels, and pnvided insight into specific characteristics 
of the reform and levels of support within the system h r the reform efforts. 

Expanded access to reform efforts and/or participants. The context of reform at a few sites 
required some researchers to expand the circle of informants. Former faculty members 
associated with locally developed reforms were interviewed at two sites to provide history and 
perspective on the reform. Researchers sat in on teacher planning meetings, faculty meetings, 
inservice training, parent meetings, meetings with visitors, and/or regional conferences. 

Two researchers played participant observer roles. One researcher became pan of the teaching 
team, sharing planning sessions and woildng with students. The other served as a 
knowledgeable assistant to the teacher, answering students’ questions and facilitating their 
thought processes through questioning techniques to elicit student thinking. 

Unique documentation. A broad range of unique documents were gathered at many sites to 
facilitate understanding of the reform and its context. At the thinking skills sites mission 
statements and philosophies demonstrated the way in which the reform was embedded in the 
larger context of the school. Teacher awareness of the personal nature of the change process, 
at one site, provided a journal documenting personal development within the reform effort. 

Each individual case study highlights its data collection process and the unique activities or 
documents associated with the development of that case. For specific data collection information 
see the report of the particular case study. 

Data Analysis - 

Data analysis took part in two phases: the individual analysis for each of the nine case smdies 
and a cross-site analysis across the nine case studies. 

Nine case studies. In analyzing their data, the case study researchers generally followed the 
analytical methodology of Spradley (1979). These steps include domain analysis constructed 
from each data source and taxonomic analysis used to derive prominent themes about the site. 
Particular descriptions and direct quotation of informants set the tone and paint the picture of 
the site. Diagrams and interpretative commentary provide explanations and connections within 
the analysis. Some researcbm developed vignettes to convey perceptions of informants and to 
give a real sense of the critical incidents and interactions at the sites. Through these 
methodologies, an attenq>t was made to provide a comprehensive, objective account. 

Cross site analysis. The actual cross-site analysis process itself included the following eight 
phases (after Rossman, 1992), the fu^ two of which were completed in the process of defming 
and conducting the nine case studies. 
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Phoit 1: Beginning. This phase entails locating the areas of interest that will provide the focus 
of the cross-site analysis. 

Phase 2: Bounding the scope. "This phase places initial boundaries on the scope of the 
synthesis." 

Phase .3: Inventorying the cases. This phase requires describing the following: 

Focus . Describe the focus such as "the curriculum itself, implementation strategies, 
instructional practices, student outcomes, professional development, the culture of the 
classroom, professiooal associations or state agencies. The researcher should identify the 
primary focus of each case study and list it, pertiaps in a matrix to facilitate comparison 
among the cases." 

Goals . Identify and list each study’s goals to help "the researcher understand the study’s 
implications a^ potential parallels with others." 

Scope . Identify the level of analytic interest, e.g., individual or nation, as well as the 
intensity of data collection and its duration. 

Complexity . Identify the complexity, e.g., the number of classrooms involved and the 
mix of research methods. 

Organization . Is the case organized "temporally, thematically, by individuals (students, 
teachers), or by some other means. These structures are part of the conceptual 
framework of the study and shape the conclusions presented." 

Audience . Who is the audience for the report? 



Phase 4: Reading the cases. "This phase entails immersion in the cases-repeated reading and 
reflecting on the texts. " 

Phase 5: Developing an interpretation of each ease. "After immersion in the set of cases 
under consideration, the researcher focuses on each case in turn to identify the key meuq)hors 
that illustrate the central meaning(s) of the cases. This may be driven by a conceptual fram^ork 
constructed prior to beginning the syntheses (always held tentatively, however) or may be more 
purely inductive ... lists of metaphors, concepts, a^ themes as expressed in phrases or vignettes 
are useful to construct. These represent the interpretations of the cases. The challenge here is 
to develop interpretations sufficiently general to comparable to the other cases yet grounded 
in the details of the specific case." 

Phase 6: Juxtaposing the eases. "Here analogic reasoning comes to the forefront as the 
researcher compares and contrasts the various tnterpretations. One case is like another in what 
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ways? different? extends and elaborates? How do the central metaphors relate to one another 
both within the cases and across cases? Which meti^hois provide ^ most explanatory power 
to capture the essences of the cases? Which metaphors most cogently, elegantly, and 
economically describe the set of cases? This process entails conq)aring themes, metaphors, and 
explanatory stories across cases. During this phase, it is likely that ccn^aring tte interpretations 
will lead to new insights into the cases~a reconceptualization of the entire work. Here, as in 
Phase 2, the use of matrices can enliven the process. ” 

Phase?: Synthesizing the cases. We are no longer dealing just with observables [cases] but 
also with unobservables [interpretations], and are connecting tte two with successive layers of 
inferential glue.* This ’inferential glue* is the stuff of synthesis, a grounded theory of the subject 
that tells us something new while preserving the sometimes contradictory specifics of the cases. " 

Phase 8: Writing the synthesis. "This fmal phase entails writing the synthesis which should 
be true to the original purpose of the woik, crisp in style, loyal to the details of the cases, but 
provide a more complex understanding of the subject than does any single case. " (Rossman, 
1992) 

Systemic perspective. A major theme of this analysis was the interconnectedness of the many 
components. The conceptual framework for the cross-site analysis was based on a 9 cell matrix, 
but the analysis was not limited to these categories; interactions across these cells were explored. 
Maintaining a systemic perspective was essential and models based on the work of Senge were 
examined for this reason. 
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